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ABSTRACT 



This paper presents the results of a development and 
psychometric evaluation of the Addiction Counselor Evaluation Scale (ACES) , 
an instrument designed for the evaluation and supervision of entry level 
Alcoholism and Other Drug Abuse (AODA) counselors based on the Technical 
Assistance Publication Series (TAP) 21. The domains of the competence of an 
AODA counselor were delineated by TAP 21, which lists 124 competencies in 4 
transdisciplinary foundations and 13 practice dimensions and subdimensions. 
The use of ACES in training, supervision, and evaluation suggested that there 
would be a value in using it as both a self evaluation tool and as a 
supervisor-completed evaluation tool. To evaluate the psychometric properties 
of the instrument student data and intern and supervisor data were collected. 
Results indicated strong test-retest and internal reliability for student and 
intern self evaluations and for supervisor-completed evaluations. Student 
self evaluations demonstrated construct validity in the form of convergent 
and discriminate validity. Construct validity for intern self evaluations and 
supervisor-completed evaluations was also consistent. These results support 
the use of self ratings and supervisor ratings to evaluate the intern or the 
training program. In addition, the use of the ACES to structure feedback and 
discussion of intern performance may enhance the training experience and 
subsequent competence of AODA counselors . (Author/ JDM) 
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Abstract 



The purpose of this poster is to present the results of the development and psychometric evaluation of the 
Addiction Counselor Evaluation Scales (ACES), an instrument designed for the evaluation and supervision of 
entry-level AODA counselors based on the Technical Assistance Publication Series (TAP) 21 (US DHHS, 1998). 
The domains of competence of an AODA counselor were delineated by TAP 21 which lists 124 competencies in 
four transdisciplinary foundations and thirteen practice dimensions/subdimensions. The use of the ACES in 
training, supervision, and evaluation suggested that there would be a value in using it as both a self-evaluation 
tool and as a supervisor-completed evaluation tool. To evaluate the psychometric properties of the instrument, 
two sets of data were collected: student data and intern and supervisor data. Results indicated strong test-retest 
and internal reliability for student and intern self-evaluations and for supervisor-completed evaluations. Student 
self-evaluations demonstrated construct validity in the form of convergent and discriminant validity. Construct 
validity for intern self-evaluations and supervisor-completed evaluations was also consistent. These results 
support the use of self-ratings and supervisor ratings to evaluate the intern or the training program. In addition, 
the use of the ACES to structure feedback and discussion of intern performance may enhance the training 
experience and subsequent competence of AODA counselors. 
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Scheidt, D. (2001, August). Addiction Counselor Evaluation Scales: TAP 21 Applied. Poster presented 
at the annual meeting of the American Psychological Association, San Francisco, CA. 

Addiction Counselor Evaluation Scales: TAP 21 Applied 

Treatment providers in the field of substance related disorders are often minimally trained, entry-level 
practitioners. For example, the Alcoholism and Other Drug Abuse (AODA) Counselor credential of the 
International Certification & Reciprocity Consortium/ Alcohol & Other Drug Abuse, Inc. (ICRC/AODA, 1999) 
requires only 270 clock hours of training and 6000 clock hours of supervised work experience prior to 
examination. Most state certifications are similar to the ICRC/AODA credential. These education requirements 
may be met in continuing education formats or at the associate’s degree level. Nevertheless, clients rely on the 
competence of these counselors for treatment. Therefore, the evaluation of professional competence is a 
necessary part of training and supervision of these entry-level treatment providers. The purpose of this poster is 
to present the results of the development and psychometric evaluation of the Addiction Counselor Evaluation 
Scales (ACES), an instrument designed for the evaluation and supervision of entry-level AODA counselors. 

The domains of competence of an AODA counselor were delineated by Technical Assistance Publication 
Series (TAP) 21 (US DHHS, 1998). This document lists 124 competencies in four “transdisciplinary 
foundations” (i.e., understanding addiction, treatment knowledge, application to practice, and professional 
readiness), and eight “practice dimensions” (i.e., clinical evaluation [i.e., screening & assessment]; treatment 
planning, referral, and service coordination [i.e., implementing the treatment plan, consulting, & continuing 
assessment and treatment planning]; counseling [i.e., individual counseling; group counseling; & counseling 
families, couples and significant others]; client, family, and community education; documentation; and 







3 



3 



professional and ethical responsibilities. Not only is this an extensive list of domains and competencies; but also, 
TAP 21 was created by an interdisciplinary team of AODA specialists, thus supporting the content validity of the 
resultant ACES. 

The ACES was created initially to evaluate students, interns, and graduating counselors in a bachelor’s 

I 

level AODA counseling program. The ACES consisted of an outline of the AODA counselor functions identified 
by TAP 21, followed by the 124 competencies with a Likert scale for each one. The anchors on the Likert scale 
were 0, not competent, a significant weakness; 1 , developing competency, a weakness; 2, competent, adequate 
performance; 3, above competent, a strength; and 4, masterfully competent, a significant strength. The 124 
competencies that comprise the ACES were collapsed into 17 rationally derived scales based on the 
transdisciplinary foundations and practice dimensions/subdimensions. The use of the ACES in training, 
supervision, and evaluation suggested that there would be a value in using it as both a self-evaluation tool and as 
a supervisor-completed evaluation tool. Therefore, the form was created to be used in both ways. 

Prior to using the ACES to evaluate the AODA counseling program or individual AODA interns and 
counselors, there was a need to evaluate the psychometric qualities of the instrument. Two sets of data were 
collected. In December 1998, test-retest data were collected from AODA students who were asked to self- 
evaluate their competencies on the ACES. Between December 1998 and May 2001, data were collected from 
interns and their supervisors. This paper presents the results of the analyses of these data. 

Method 

Subjects. Respondents were students, interns, and intern supervisors in a bachelor’s degree program in 
AODA counseling. A total of 1 12 students in the AODA counseling classes participated in a test-retest reliability 
study, self-evaluating their competencies with the ACES twice, separated by approximately two weeks. A total 
of 80 students completed both administrations. Also, by May 2001, approximately 59 interns and 45 of their 
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supervisors have participated in this study. 

Instrument. The ACES was prefaced by an informed consent sheet, a report of overall grade point 
average and grades in each required AODA counseling course. The ACES itself is described above. 

Procedure. For the test-retest reliability study, students in AODA counseling classes were given the 
opportunity for extra-credit by self-evaluating on the ACES. Two weeks later, students who had turned in their 
self-evaluation were given a repeat administration of the self-evaluation on the ACES. 

For the intern/supervisor validity study, interns self-evaluated their competencies at two times: pre- 
internship and post-internship. These interns were evaluated by their internship supervisors on the ACES at the 
conclusion of their internship. 

Results 

AODA students: Test-Retest Reliability, Internal Reliability, and Construct Validity. Results of the test- 
retest reliability study of self-evaluation by AODA students indicated strong, significant correlations (all at least r 
> •70, E < .01). In addition, internal reliability assessed with Cronbach’s alphas for all 17 rationally derived 
subscales were satisfactory (r > .78) , with most greater than .90. Construct validity of self-evaluations was 
supported by discriminant validity such that there were few significant correlations with overall grade point 
average. Convergent validity was supported by a pattern of significant correlations between ACES scales and 
grade point average in AODA courses. See Table 1 . 

AODA Interns and Supervisors: Internal Reliability and Construct Validity of Self-Evaluations and 
Supervisor-Completed Evaluations. Analyses indicated similarly strong internal reliability when the ACES was 
used as an intern self-rating and when used as a supervisor rating of interns. Construct validity was supported by 
consistent significant correlations between self and supervisor evaluations on 16 out of 17 scales. On only one 
scale did interns differ significantly with their supervisor’s rating, with interns rating themselves lower on 
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treatment knowledge than their supervisors. 

Discussion and Conclusions 

Results indicate strong test-retest and internal reliability for student and intern self-evaluations and for 
supervisor-completed evaluations. Student self-evaluations demonstrated construct validity in the form of 
convergent and discriminant validity. Construct validity for intern self-evaluations and supervisor-completed 
evaluations was also consistent. These results support the use of self-ratings and supervisor ratings. For example, 
these ratings may be used to evaluate the intern or the training program. In addition, the use of the ACES to 
structure feedback and discussion of intern performance may enhance the training experience and subsequent 
competence of AODA counselors. 

Use of this instrument for evaluation of post-training counselors requires additional study. Demand 
characteristics of performance evaluation may affect the validity of the evaluation. Therefore, the validity of the 
ACES with post-training counselors requires empirical evaluation. 
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Table 1. Psychometric Qualities of the Addiction Counselor Evaluation Scales (ACES): Student Sample: Self - 
Evaluation 



Competency 


Test- 
Retest r 
(n = 84) 


Cronbach’s 
alpha 
(n = 112 ) 


Convergent 
Validity 
r with major 
GPA 
(n = 46) 


Discriminant 
Validity 
r with overall 
GPA (n = 

98) 


A. FOUNDATIONS FOR ADDICTION PROFESSIONALS 






I. UNDERSTANDING ADDICTION 


85*** 


.86 


.58*** 


-.02 


II. TREATMENT KNOWLEDGE 


.72** 


.78 


.55*** 


-.07 


III. APPLICATION TO PRACTICE 


79 *** 


.92 


.40** 


_ 25 ** 


IV. PROFESSIONAL READINESS 


gj *** 


.86 


37 ** 


-.05 


B. ADDICTION COUNSELOR COMPETENCIES 






1. Clinical Evaluation 










Screening 


g4*** 


.97 


44 * * 


-.17* 


Assessment 


75 *** 


.92 


.56*** 


-.06 


2. Treatment Planning 


go*** 


.98 


47 * * * 


-.11 


3. Referral 


g4*** 


.97 


.46** 


-.16 


4. Service Coordination 










Implementing the Treatment Plan 


79 *** 


.97 


.50*** 


-.12 


Consulting 


g3*** 


.91 


51 *** 


.04 


Continuing Assessment and Treatment 
Planning 


g7*** 


.97 


.39** 


-.10 


5. Counseling 










Individual Counseling 


g3*** 


.98 


.31* 


-.11 


Group Counseling 


g3*** 


.96 


.04 


-.11 


Counseling for Families, Couples, and 
Significant Others 


g9*** 


.96 


.35** 


-.07 


6. Client, Family, and Community 
Education 


.85*** 


.96 


40** 


-.08 


7. Documentation 


91 *** 


.97 


.38** 


-.13 


8. Professional and Ethical 
Responsibilities 


85*** 


.96 


49*** 


-.11 
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Table 2. Psychometric Qualities of the Addiction Counselor Evaluation Scales (ACES): Intern Sample: Self & 



Competency 


Cronbach’s 
alpha: 
Intern 
(a =59) 


Cronbach’s 
alpha: 
Supervisor 
(a = 45) 


Convergent 
Validity 
r with 
supervisor 
evaluation 
(a = 41) 


Paired t 
test: Self - 
Supervisor 
Ratings 
(df =41) 


A. FOUNDATIONS FOR ADDICTION PROFESSIONALS 






I. UNDERSTANDING ADDICTION 


.81 


.87 


.36* 


0.17 


II. TREATMENT KNOWLEDGE 


.74 


.87 




-2.03* 


III. APPLICATION TO PRACTICE 


.89 


.93 


.42** 


-0.94 


IV. PROFESSIONAL READINESS 


.87 


.93 


.34* 


0.01 


B. ADDICTION COUNSELOR COMPETENCIES 






1. Clinical Evaluation 










Screening 


.91 


.97 




0.41 


Assessment 


.90 


.90 


.36** 


0.48 


2. Treatment Planning 


.95 


.97 


.45** 


- 0.22 


3. Referral 


.94 


.96 


.24 


-0.49 


4. Service Coordination 










Implementing the Treatment Plan 


.89 


.95 


.38** 


-1.44 


Consulting 


.90 


.92 


47 *** 


-0.95 


Continuing Assessment and Treatment 
Planning 


.92 


.95 


.36** 


-1.05 


5. Counseling 










Individual Counseling 


.96 


.98 


52*** 


0.50 


Group Counseling 


.89 


.96 


27 ** 


-0.14 


Counseling for Families, Couples, and 
Significant Others 


.93 


.95 


39 ** 


-0.62 


6. Client, Family, and Community 
Education 


.93 


.97 


49 * * * 


-0.83 


7. Documentation 


.89 


.94 


.42** 


-0.25 


8. Professional and Ethical 
Responsibilities 


.90 


.96 


.60*** 


0.26 
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